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Education reform in the United States is increasingly driven by data. There are important 
national efforts - including federal funding - to help states develop sophisticated 
longitudinal data systems that track students’ academic history and performance from 
their first entry into the school system through their college and even graduate school 
careers. In anticipation of the ever-greater sophistication of these systems and the 
growing accuracy and detail of the data they generate, policy-makers throughout the U.S. 
have in effect established a nationwide education accountability system that relies on 
these data as the principal measure of the performance of students, teachers, and 
schools. And in response to the level of performance thus determined, there are made a 
whole host of data-based decisions: about remediating, restructuring, or closing schools; 
about rewarding, promoting, re-assigning, and educating or re-educating teachers and 
school administrators; about promoting, retaining, re-assigning, or supporting students; 
even about the nature of the curriculum and instruction in the classroom. 

As a research and technical assistance organization, CNA Education would be the last to 
deny the importance of good data and the first to applaud efforts to improve the quality 
and scope of the data available. Solid data are critical to CNA’s mission to further general 
understanding of the efficacy of specific education reform initiatives and to address the 
needs of our district-level and state-level customers and constituents for a clearer 
assessment of their own specific efforts and of the available options for improving their 
education outcomes. The three papers in this collection make the strong case, however, 
that the availability of good data is not all that policy-makers and educators need for their 
decisions. In addition, they need to understand the limitations of their data and how to 
use and interpret them appropriately. And they must ask - and answer to their 
satisfaction - a number of conceptual and technical questions that have implications for 
the validity, interpretation, and very meaning of the data they seek to use. 

It is our experience at CNA Education that the ability of state and district education 
leaders to address these questions adequately depends upon a capacity for customized 
research and analysis that they often do not have. And that experience is a major 
motivation for the production and dissemination of these three papers. Whether that lack 
of analytic capacity is a function of chosen priorities, scarce funding, or a combination of 
both, we at CNA believe it is imperative that states find a way to redress it if they are fully 
to exploit the promise of their ambitious data systems and make policy and practice 
decisions that are properly informed by strong data, rigorous research, and thorough 
analysis. And we believe that an important opportunity for states and districts to meet 
their needs for adequate analysis lies in taking advantage of the external capacity that 
CNA and other education research-oriented agencies have to offer. The three papers 
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attempt to demonstrate the nature and added value of that extra capacity, each by 
discussing a different kind of analysis that can be contributed. 

The first paper, The State Data Analysis Gap: A Threat to Education Reform, by Arthur 
Sheekey and Michael Allen, focuses squarely on the motivating concern for the series, 
that the tremendous growth in the power of state education databases and in our reliance 
upon them are not matched by a similar growth in the capacity of states to provide the 
depth and breadth of analysis that are required to take full advantage of the data that are 
increasingly available. The ability of states to use the data at their disposal for serious 
research purposes beyond immediate administrative and reporting needs is seriously 
compromised by priorities that mirror their historic focus on reporting and ensuring 
compliance and also by states’ current fiscal predicament. The lack of research capacity 
not only compromises states’ ability to get the most out of their data systems, it also 
poses a threat to the continued quality of those systems once the federal funding for their 
initial development is exhausted.. The paper offers several examples of how inadequate 
research and analysis can lead to a misdiagnosis of problems and thus to misdirected 
policy solutions. It concludes by suggesting that the federal government should make 
specific provisions in the renewal of key education programs that encourage state 
agencies to focus more on research and analysis. And it discusses the potential 
contribution of the Regional Education Laboratories (RELs) to a more vigorous state 
research effort. One of those agencies, REL Appalachia, is operated by CNA Education. 

The second paper, written by Kyle Southern and Joe Jones, is entitled Getting State 
Education Data Right: What We Can Learn from Tennessee. It discusses four principles 
that should guide states’ use of data and gives an example that illustrates the importance 
of each. The four principles are (a) establish common definitions of terms; (b) anticipate 
potential unintended consequences of data definitions and priorities; (c) ensure that data 
definitions are applied consistently; and (d) disaggregate data in order to reveal the most 
complete and accurate picture. As the paper makes clear, these challenges are both 
technical and political. Consistent with the theme of the series, the paper notes that 
meeting these challenges requires a detached objectivity, and at times an in-depth 
research capacity that local stakeholders and agencies may not be able to provide. The 
paper points out that the high stakes consequences that are increasingly attached to data 
make successfully addressing the challenges both more essential and more charged 
politically. On the one hand, inaccurate or unreliable data lead to a misdiagnosis of 
problems and thus to ineffective solutions in response. On the other hand, data that are 
sufficiently accurate and fine-grained may highlight uncomfortable problems that were 
previously invisible but now no longer can be ignored. The paper concludes that meeting 

Three Perspectives on Getting Data Right 
2 



these data challenges requires a combination of solid research and analytic capacity, 
political will, and a commitment not to act precipitously before assessing the strengths 
and limitations of the data available. 

The final paper, Michael Allen’s The Equitable Distribution of Effective Teachers: Can 
States Meet the Research Challenges Required for Success?, describes the analytic and 
research groundwork that states must undertake in order to address one of the most 
important requirements of current federal education policy. Although virtually all states 
understand the need for such an analytic effort, their ability to muster it and to engage in 
an ongoing, rigorous assessment of the success or failure of their various efforts to 
achieve equitable teacher distribution depends upon a level of research expertise and 
capacity that may well not reside in state education agencies. In particular, the paper 
identifies three kinds of analytic tasks that are ultimately relevant not only to equitable 
teacher distribution but to any education reform priority: (1) clarifying key terms and 
concepts; (2) understanding the implications of the relevant existing research; and (3) 
grasping clearly the current state education picture and developing a reliable projection 
for the future. As an example of what could be possible, the paper cites the extensive 
research capacity Tennessee has developed to accomplish its equitable distribution 
goals and carry out other aspects of its ambitious reform agenda. The paper cautions, 
however, that the resources available to Tennessee - both its premier institutions of 
higher education and its substantial funding as a Race to the Top grant recipient - are 
not replicated in all states. Other states may have to rely more heavily on the kind of 
expertise that independent research entities, including the RELs, can provide them. 

The emphasis on the role of the RELs in the three papers is not coincidental but a 
product of CNA Education’s experience operating REL Appalachia, which serves the 
states of Kentucky, Tennessee, Virginia, and West Virginia. As the Sheekey paper 
specifically notes, realizing the full potential of the RELs to contribute substantially to the 
kind of research and analytic support that will serve the states most effectively will require 
a more vigorous commitment to the RELs’ quick response role on the part of the Institute 
of Education Sciences, the U.S. Department of Education agency that governs them. 
There is an inherent tension in such a commitment because it appears to conflict with the 
charge to the RELs to produce experimental or quasi-experimental studies that can 
establish causal connections between education interventions and student outcomes. To 
be sure, such rigorous studies play a vital role in establishing the effectiveness of various 
curricula, instructional strategies, or policies, and they thus provide policy-makers and 
education leaders with valuable information. Such studies require a long-term research 
effort, however, and thus they cannot inform many of the more immediate policy and 
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instructional decisions that state leaders must make. These can include, for example, a 
comparison of a state’s policies on retention in grade with those of other states, a prompt 
assessment of the adequacy and validity of available data about teacher retention and 
attrition, or perhaps an analysis of what is known about the effectiveness of various 
strategies currently used to decrease high school dropouts. 

The challenge for the RELs, then, is to meet the more immediate applied research and 
analysis needs of their customers and stakeholders while maintaining high standards for 
methodological rigor. Such an effort always will appear as a regrettable compromise so 
long as we insist upon a hierarchical view of the range of research methods that places 
experimental research at the top and descriptive research at the bottom. - especially if 
that description is qualitative. If, instead, we consider the menu of available research 
methods to be a spectrum, with each method being justified by its ability to address 
satisfactorily the research requirements of situations appropriate to its application, then 
the taint of compromise is dispelled. 

The three papers presented here are not per se about research methodology. Taken 
together, however, they re-enforce the importance of ensuring that there is adequate 
capacity to minister more readily to the non-experimental research needs of policy- 
makers and educators that at present are too frequently unmet or even overlooked. We 
hope you find these papers both informative and thought-provoking, and we welcome 
your comments and your suggestions for related further issues that we might explore. 

CNA Education is a program of CNA that operates within the Institute for Public 
Research, one of the corporation’s two principal divisions. The I PR is the civil counterpart 
to CNA’s Center for Naval Analyses, a federally funded research and development center 
for the U.S. Navy and Marines. The culture of CNA derives from our more than 60-year 
history of bringing multidisciplinary research methods to issues of national importance. 
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